CONNECTING THE CLASSROOM TO CAREERS: THE STATE’S ROLE IN EXPANDING WORK-BASED LEARNING 


" The Connecting the Classroom to 
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the state’s role in expanding high- 
quality work-based learning 
opportunities for all students, with a 
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Measuring Work-based Learning major barriers at the K-12 level. 
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leading state approaches, as well as 


THE STATE’S ROLE IN WORK-BASED key policy questions and relevant 


resources that state and local leaders 


LEARNING should consider as they examine their 


own systems. 


State Leaders Connecting Learning to Work 


As the nation’s education leaders and employers seek to work together 
to create a well-prepared, competitive workforce, work-based learning — 
a strategy that has existed for decades — is back in the spotlight as an effective strategy for connecting students’ 
classroom learning to their future careers. 


With this renewed interest, the work-based learning of today is being transformed as a means to build on students’ 
academic experience and career interests, rather than simply to release them from the school day for work that is 
unconnected to their education. Work-based learning can and should be available to all students regardless of whether 
they are enrolled in a Career Technical Education (CTE) program of study or not. What is most important is that the 
experience is informed by workplace standards and is connected to students’ classroom learning as a means to provide 
context and relevancy.’ 


What is Work-based Learning? 


Work-based learning is an educational strategy that offers students an opportunity to reinforce and deepen their 
classroom learning, explore future career fields and demonstrate their skills in an authentic setting. 


This series defines work-based learning as a continuum of experiences that helps prepare students for 


postsecondary education and careers. High-quality work-based learning should begin in the early grades with 
activities that help build students’ awareness of possible careers. This exploration continues through middle and 
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more intensive career preparation activities such as school-based enterprises, internships and pre-apprenticeships 
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THE STATE’S ROLE IN MEASURING WORK-BASED LEARNING 

One of the most important — yet perhaps most challenging — roles that states play in work-based learning is 
measuring and evaluating program quality. While many programs are designed and operated at the local level, a 
strategic data collection and evaluation plan can help states ensure program quality, identify and scale successful 
programs, and share promising practices. 
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The data that a state collects may depend on the type of work-based learning program it wishes to evaluate. For low- 
touch activities such as job shadowing or mentoring, it may suffice to measure the number of students who participate 
inthe program. Descriptive metrics such as this illustrate the breadth but not the quality of the program. For more 
intensive experiences such as internships or apprenticeships, states may want to consider a holistic evaluation of the 
program’s quality that may include measures of student skill gain and the degree to which the program aligns to 
industry standards. 


States can measure work-based learning in two ways: a systems-level approach and a student-level approach. A 
systems-level approach examines program data such as student participation, industry alignment and adherence to 
occupational standards to assess the quality of and equitable access to programs offered. This can allow states to 
gauge the breadth and quality of work-based learning, help employers understand the competencies of students 
graduating from such programs, and serve as an indicator of career readiness. 


In a student-level approach to measuring work-based learning, the student, rather than the program, is the unit of 
analysis. Relevant indicators may include demographics of student participants; attendance rates; and student skill gain 
as measured through assessments, student portfolios or employer evaluations. A student-level approach often presents 
more obstacles, as it requires states to design a valid and reliable system to assess student learning and collect data ina 
consistent way across districts. 


While both the systems-level and the student-level approaches involve different units of analysis, they are not mutually 
exclusive. In fact, states should consider using both approaches simultaneously. As states work to develop new 
accountability systems under the Every Student Succeeds Act (ESSA), some are considering career readiness indicators 
that include both systems-level and student-level measures of work-based learning. However, measuring work-based 
learning and ensuring program quality is a critical step before scaling a program statewide or valuing it within an 
accountability system. 


While work-based learning has been offered formally and informally at the local level for decades, most states are new 
to measuring work-based learning. In the spirit of “that which gets measured gets done,” states can promote high- 
quality work-based learning through a strong data collection and evaluation strategy. 
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KEY QUESTIONS FOR MEASURING WORK-BASED LEARNING 


Creating a Strategy 


Does your state already collect data on work- 
based learning? If so, what kind of data are being 
collected and how are they being used? Does the 
population include CTE students, CTE 
concentrators or all students? 


What does your state want to achieve through its 
work-based learning measurement strategy? Is 
the goal to monitor and learn from work-based 
learning activites or identify and improve 
program quality? 

Should this goal be differentiated for activities 
along the work-based learning continuum? If so, 
how are data measured differently for each 
activity? For example, what data are collected for 
job shadowing as compared to internships? 


Does your state have a definition of work-based 
learning? Is this definition codified in state 
statute? Is this definition used consistently across 
state agencies and at the local level? 


Are state policies in place that could be used to 
develop and scale a work-based learning 
measurement strategy? 


Collecting the Data 


Are there state policies in place that regulate data 
collection activities? Are there privacy regulations 
that could complicate data collection and 
evaluation? 


What data collection infrastructure, such as the 
state longitudinal data system, is already being 
used at the state and local level? Are data-sharing 


agreements in place with local schools and 
districts? 


Can existing data infrastructure be expanded to 
include work-based learning? 


What can your state do to ensure reliable data 
submissions? Are there incentives or penalties 
available to motivate compliance? 


How do you ensure that data are consistent and 
reliable across the state? What technical 
assistance is provided to support consistent data 
collection? Are there checks in place such as 
monitoring and auditing data collection activities 
to ensure consistency and validity statewide? 


Evaluating Program Quality 


How does your state define student-level success 
for work-based learning? Program-level success? 
What must be present for a program to be 
considered “high quality?” 

How does your state’s data collection strategy 
connect to your definition of work-based 
learning? Are you measuring program outcomes 
(such as skill gain or employment) or only outputs 
(such as participation)? 


What role does business and industry play in 
evaluating work-based learning programs at the 
state and/or local level? 


How can evaluations be used to create a 
feedback loop and ensure continuous program 
improvement? What is being done to target and 
assist programs in need of improvement? What is 
being done to strengthen or phase out low- 
quality work-based learning programs? 
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ENSURING PROGRAM QUALITY 


High-quality work-based learning opportunities can provide 
students with authentic, real-world experiences that are 


What Is a High-Quality Work-based Learning 
Experience? 


Tennessee uses the following 12 components to define a 
high-quality work-based learning experience: 


aligned to industry practices and prepare them for future 
careers. Yet, quality can be defined differently depending on 
the type of program. While short-term, exploratory activities 
such as job shadows and career days are designed to 
increase student awareness about career opportunities, 
intensive work-based learning experiences such as 
internships and youth apprenticeships should involve an 
element of demonstrable skill gain. A primary function of a 
work-based learning measurement strategy is to define and 
differentiate intended outcomes for different types of 
experiences, and then use those outcomes to ensure that 
programs are high-quality and meaningful for students. 


WEST VIRGINIA’S SIMULATED WORKPLACE 


West Virginia employs a systems-level approach to 
measuring and evaluating the quality of its work-based 
learning program, Simulated Workplace. By leveraging third- 
party evaluations from industry partners, the West Virginia 
Department of Education not only showcases its CTE 
programs to local industry, but is also able to improve the 
quality and authenticity of work-based learning for 
participating students. Creating an authentic work 
environment that meets industry standards allows students 


A purposeful focus on applied learning in preparation 
for postsecondary education and careers 

Learning outcomes as the driver for designing 
experiences and Personalized Learning Plans 
Relevance to student interests, their plan of study and 
learning goals 

Integration with curriculum or connection to related 
instruction 

Sufficient variety to provide exposure to multiple 
career options 

Sufficient depth to allow for employability skill 
development and professional community 
engagement 

Ongoing interaction with professionals from industry 
and the community 

Close supervision from both teachers and employers 
Opportunities for reflection and analysis 


. Assessment of student learning that is aligned with 


industry-specific expectations 


. Alignment with postsecondary and career 


opportunities regionally 


. Documentation of student learning through the 


development of artifacts and portfolios 


to develop the technical and professional skills necessary for future success. 


The Simulated Workplace program was launched in 2013 after industry leaders expressed a need for students to learn 
professional skills — such as punctuality, teamwork and maintaining safe workspaces - in addition to the technical skills 
typically taught in CTE classrooms. Simulated Workplace was initially designed for CTE programs and is operated under 
the Division of Career Technical Education, though the state says there is increasing interest in expanding the program 
to academic classrooms. After a four-year pilot and roll-out at an increasing number of high schools throughout the 
state, the program was scaled statewide in 2015. At the same time, the West Virginia Board of Education voted to adopt 
12 Simulated Workplace protocols, which govern the design of the programs and ensure consistency and quality. 
During the 2015 school year, over 13,000 students participated in more than 500 Simulated Workplace classrooms 
across West Virginia. 


Through the Simulated Workplace program, students transform their classrooms into businesses to create an authentic 
environment where they can develop and practice both technical and professional skills. Participants in the program are 
treated like employees: they are required to pass an interview for entry into the class, fill assigned roles within the 
company, participate in random drug tests, write a company handbook and pass a safety training. Students also 
participate in an industry evaluation where inspectors, who are employers from the field, observe and rate programs 
based on their adherence to industry standards — not unlike an actual business. 
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While industry evaluations provide specific feedback to students and teacher facilitators, they also give the state an 
opportunity to evaluate the quality of a specific program and the degree to which it aligns with industry norms. The 
evaluation is coordinated by the 
Department of Education, which 


recruits employers to visit the 
classroom as inspectors, schedules “Classroom/lab area represents an applicable workplace environment” 


Sample Questions from the Industry Evaluation Rubric 


site visits, and even provides an 


“Proper industry safety signage displayed” 
Industry Evaluation rubric that 


inspectors can use to assess Equipment and tools are up-to-date to meet occupational standards 
Simulated Workplace programs. The “Students can discuss how acquired program skills will assist in 
department's Industry Evaluation furthering their education and career” 
rubric was designed collaboratively 
with private sector and higher ae ———_—_—————— 
education leaders and includes Green - Outstanding, meets industry standards in all areas and is a model for others. 
occup ational norms fors afety, Amber/Green = - Meets industry standards in most areas but not quite at industry standards. 

; : Amber/Red - Meets some industry standards, but needs improvement. 
equipment and workplace behavior. Red - Does not meet industry standards and requires significant improvement. 


The industry evaluation also provides 
a summative assessment of program 
quality that the state can use to identify programs in need of improvement. Simulated Workplace programs that receive 
an evaluation score of 85 percent or higher are recognized by the Department of Education as an “Industry Endorsed 
Program.” However, if a program receives a rating lower than 85 percent, the Department sends a technical support 
team that assists students and administrators in developing a program improvement plan. Once a program develops an 
improvement plan and addresses the issues surfaced in the industry evaluation, it can request an additional review. If a 
program continues to fail the evaluation, the state has the option to not approve the program for state or federal 
dollars. This cycle of evaluation and technical assistance creates a feedback loop that allows the state to highlight 
quality programs, support those in need of improvement and phase out low-quality programs. 


HOW TENNESSEE ENCOURAGES CONTINUOUS IMPROVEMENT AT THE LOCAL LEVEL 


In Tennessee, a systems-level approach to program evaluation and improvement is part of the state’s new framework 
for work-based learning. The framework, which the State Board of Education adopted in 2014 as part of a wave of 
comprehensive reforms under Governor Bill Haslam’s Drive to 55 campaign", requires local districts to evaluate work- 
based learning programs and to strive for continuous improvement. While Tennessee is still in the early stages of 
collecting and evaluating data at the state level, the state has begun a process of measuring work-based learning by 
supporting local districts as they monitor and evaluate their own programs at the local system level and student level. 


Tennessee’s framework outlines the core components and expectations for activities, such as a credit-bearing work- 
based learning capstone. The capstone — which can be an apprenticeship, cooperative education, internship, or 
immersive classroom-based experience — is taken as a course within a student's program of study and allows the 
student to apply academic and technical knowledge to the workplace. Student portfolios demonstrate standard 
attainment and employability skills to ensure depth of experience and learning. While Tennessee delegates much of the 
coordination and administration of work-based learning to local school boards, it requires them to adopt processes for 
evaluation and assessment, based on acommon set of state-developed standards, to ensure the experiences they 
provide are high quality. 
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To support local school boards in this work, the 
Department of Education provides guidance in the form 
of a toolbox of evaluation resources and the Work-based 
Learning Implementation Guide, which describes how 
local school boards should evaluate program quality and 
promote continuous improvement. Programs must align 
with the work-based learning evaluation policies 
articulated in the framework for districts to offer credit 
for the capstone course. 


The Department encourages local school boards to 
include industry partners in the evaluation process and 
even provides a sample employer satisfaction survey for 
program coordinators to use. This resource aims to 
identify ways in which to improve the experience for 
industry partners as well as gauge the engagement and 
preparation of students entering the program. Further, 
the state’s Work-based Learning Framework requires 
local boards to adopt a “continuous improvement” 
process to engage stakeholders and ensure that 
programs are sustainable and aligned to workforce 
needs. Local boards can access program evaluation 
frameworks in the toolbox that allow them to reflect on 
and identify opportunities to improve work-based 
learning. 


While Tennessee is in the initial stages of building the 
data infrastructure to monitor programs at the state 
level, the state is designing and implementing a three- 
tiered evaluation and assessment strategy to determine 
the regional alignment of work-based learning programs 
to workforce needs, quality program measures for 
district implementation, and student growth measures 
to demonstrate learning. Ultimately, the state plans to 
use information gathered through this process to 
evaluate student participation in work-based learning, 
alignment of programs to regional employment needs 
and provide targeted guidance to districts. 


How Other States Collect Work-based Learning Data 


Just getting the processes and protocols off the ground 
can be the hardest part of measuring work-based 
learning (WBL). The best place to start is by creating a 
consistent and measureable definition and integrating 
it into existing data collection systems. Here are some 
examples of other states that have already begun this 
work: 


" Virginia collects WBL data using eight designated 
course codes in its CTE Reporting System. 
Categories of WBL experience include Cooperative 
Education, Youth Apprenticeship, Internship, 
Mentorship, Job Shadowing, Service Learning, 
Clinical Experience and Supervised Agricultural 
Experience. Submissions through the CTE Reporting 
System allow Virginia to monitor different types of 
WBL at the state level. 


In South Carolina, the Education and Economic 
Development Act requires schools to collect and 
report WBL participation through the state’s school 
data system, which includes 10 state-approved 
categories of WBL, ranging in intensity from job 
shadowing to apprenticeships. Data are then 
aggregated and included in school, district and state 


report cards. 


Georgia maintains a work-based learning database 
called C-NET, which captures training plans, student 
progress and employer information for each WBL 
opportunity. This database allows administrators 
with the Department of Education and local system 
employees to access aggregated reports on WBL 
participation. 


MEASURING STUDENT OUTCOMES THROUGH CONNECTING 


ACTIVITIES IN MASSACHUSETTS 


States can also measure student outcomes to ensure work-based learning programs are preparing students for the 
workforce effectively. A student-level approach allows the state to monitor student progress and success, which in turn 
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can help the state identify challenges in program design or opportunities to scale a successful program. As useful as 
student-level data collection is, however, many state education agencies do not have the infrastructure or means to 
determine student outcomes from work-based learning programs. One innovative strategy comes from 
Massachusetts, where the Department of Elementary and Secondary Education uses a skill assessment tool to collect 
and report student work-based learning performance data. 


Connecting Activities Performance Measures 


During Fiscal Year 2015, Connecting Activities 
generated the following outcomes: 


10,487 students were placed in internships at 
3,477 employer sites 

Employers invested $14,690,000 in wages to 
support student internships, almost a 5:2 
match (the legislation requires a 2:1 match, 
but the initiative regularly exceeds that) 
7,677 (73 percent) students used the 
Massachusetts Work-Based Learning Plan on 
the job 

7,677 students participated in 
classes/workshops including career 
exploration, work readiness and internship 
workshops 

2,775 employers sponsored career awareness 
and exploration activities for students 
including career days, job shadowing and 
guest speaker programs 

185 high schools were partners in the 
Connecting Activities initiative and another 50 
were members, accounting for more than half 
of the state’s 393 public high schools 


BUILDING A WORK-BASED LEARNING PLAN 


Massachusetts provides one example of how states can 
evaluate and collect student-level data for a well-defined 
work-based learning experience. Although Massachusetts 
collects and examines systems-level data as well, the state is 
particularly interesting for its approach to measuring student- 
level learning outcomes. 


In Massachusetts, all students can access work experience and 
career development opportunities through its School to Career 
Connecting Activities Initiative (Connecting Activities). 
Originally piloted through a five-year federal grant, 
Connecting Activities was formally launched in 1997 and has 
continued to be supported through a dedicated line item in the 
state budget. The program is operated by 16 local Workforce 
Development Boards that each work as intermediaries to build 
public-private partnerships between schools and employers. 
These boards support career exploration through internships, 
job shadow days, career days, employer guest speaker 
programs, workshops, teacher externships and curriculum 
development. In 2015 alone, about 10,500 students — most of 
whom were enrolled in comprehensive high schools, not the 
Commonwealth's full-time, regional CTE high schools — 
participated in internships through Connecting Activities. 


Program staff and employers work together to structure the 


experience around the Massachusetts Work-based Learning 


Plan, which guides student learning and connects workplace activities to specific career skills. 


The plan includes four major components: 


A job description; 


A list of foundation skills — including “work ethic and professionalism skills” such as punctuality, workplace 
appearance, and taking initiative and “communication and interpersonal skills” such as speaking, listening and 
interacting with co-workers — that are universal to all jobs and aligned to the state's definition of college and 


career readiness; 


A customized list of workplace skills that are selected by the worksite supervisor and the program coordinator; 


and 


An evaluation of the student's performance and progress. 
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For the performance evaluation, worksite supervisors assess students on their foundation and workplace skill 
competency using a five-point scale. Evaluations are completed at least twice over the course of the experience — at the 
beginning of the work experience to determine a baseline level of performance and then at the end of the program to 
measure progress and skill gain. 


Connecungactiines Agareaseed skill LEVERAGING DATA SYSTEMS TO EVALUATE STUDENT 


Gain Measures from Work-based Learning OUTCOMES 


Plans Massachusetts collects Connecting Activities data through an 

For 2014-15 school year online portal called the Massachusetts Career Ready Database, 
which is maintained by the Department. This database, which was 
launched in 2008 to support Connecting Activities programs, serves 
not only as a resource for students and employers to guide the 
work experience, but also as a mechanism to collect, monitor and 
compare data at the local, regional and state level. Work-based 
Learning Plans can be entered through the database system, 
uploaded as an electronic document, submitted by paper or 
entered through a mobile application. Entering the plan through 
the database allows students and program coordinators to track 
progress and reflect on skills learned. The database also allows 
program coordinators to view aggregated data from the Work- 
based Learning Plans to identify the foundation and workplace 
skills with the lowest growth and to focus their program design on 
developing those skills. 


Most notably, the system allows the state to calculate student 
outcomes by comparing the baseline student skills evaluation— 
which is submitted at the beginning of the program — to the final 
assessment. This evaluation provides a measure of the skills gained 
through Connecting Activities — a valuable student outcome 
measure that helps determine the effectiveness of the program. 
Massachusetts also uses the database to identify trends across the 
Connecting Activities programs and determine whether industry 
partners are meeting the legislatively mandated 2:1 employer 
funding match, which program administrators have consistently exceeded. The state compiles data into an easy-to- 
read dashboard, which it shares publicly on the Connecting Activities website. 


One ongoing challenge with the Career Ready Database is that, even though workforce boards are required to submit 
data on student participation directly to the state through the online database on a quarterly basis, the Work-based 
Learning Plan itself is not mandatory. As such, employers and program coordinators sometimes elect to use their own 
system of evaluation instead. In school year 2014-15, 73 percent of student work-based learning opportunities were 
evaluated using the Work-based Learning Plan. 


Another challenge is ensuring consistency and reliability across worksite supervisor evaluations, which are subject to 
variability. While one supervisor may consider a student to be “Proficient” at a specific foundation skill, another may 
rate the student merely “Competent.” The Department has worked to address this issue by developing a rubric that 
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describes the performance levels in objective terms and by providing guidance and training materials for worksite 
supervisors in use of the rubric. This guidance includes a short training video that explains how to evaluate students 
using the Work-based Learning Plan. 


LESSONS LEARNED 


Engage business and industry in the program evaluation process to help ensure that work-based learning 
programs are authentic and meaningful. 


As program participants, students have valuable perspective on the effectiveness of work-based learning 
programs. Involving students in the evaluation and program improvement process can provide valuable feedback 
and build student ownership. 


Do not shy away from evaluating employability skills. Although these skills can be difficult to measure, they help 
gauge student readiness for entering the workforce. Supervisor evaluations, such as those used in Massachusetts 
for the Connecting Activities Work-based Learning Plan, leverage the expertise and observations of site hosts to 
evaluate student competencies. 


While it is easier to start by collecting descriptive work-based learning data — such as the number of hours 
worked, number of host sites provided or amount of wages earned — it may be appropriate to develop new 
systems to collect data on student outcomes. Data such as skill gain demonstrate the outcomes of work-based 
learning and can indicate whether or not the program is adequately meeting its objectives. 


The work does not end when the evaluation is completed. Be sure to develop a continuous improvement process 
to push all programs to higher levels of quality and phase out low-performing programs as needed. 


RESOURCES 


Massachusetts’ Connecting Activities Guide and e Tennessee’s Work-Based Learning Toolbox, 
Glossary, FYas5 Annual Report, History and Implementation Guide and Career Practicum 
Performance Metrics Course Standards 

Training Video for Employers Sponsoring Youth e Virginia's CTE Reporting System User's Manual 


Work-based Learning Experiences through 
Connecting Activities 


e West Virginia: Simulated Workplace Operational 
Manual and Industry Evaluation Rubric 


South Carolina’s Work-based Learning Manual 
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